Skip to content

fix: grad norm check for automodel gpt oss nightly#1708

Merged
terrykong merged 1 commit intomainfrom
hemil/fix-gpt-oss-nightly
Jan 5, 2026
Merged

fix: grad norm check for automodel gpt oss nightly#1708
terrykong merged 1 commit intomainfrom
hemil/fix-gpt-oss-nightly

Conversation

@hemildesai
Copy link
Contributor

@hemildesai hemildesai commented Jan 4, 2026 •
edited by coderabbitai bot
Loading

fixes the end grad norm check after #1693

Summary by CodeRabbit

  • Tests
    • Updated metric validation thresholds in model training test suite to reflect revised performance expectations.

✏️ Tip: You can customize this high-level summary in your review settings.

@hemildesai
fix: grad norm check for automodel gpt oss nightly
7795224 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
@hemildesai hemildesai requested a review from a team as a code owner January 4, 2026 23:36
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Jan 4, 2026

📝 Walkthrough

Walkthrough

Test configuration file updated to modify gradient norm metric validation. The check for train/grad_norm at step 50 transitions from a single upper bound (< 2.5) to a bounded range (10.0 ≤ value ≤ 17.5), altering the acceptance criteria for model training validation.

Changes

Cohort / File(s) Summary
Test metric validation
tests/test_suites/llm/sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel.sh		 Modified train/grad_norm check at step 50 from single upper bound (< 2.5) to range check (10.0 ≤ value ≤ 17.5). Adds lower bound and raises upper bound threshold.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Suggested labels

CI:L1, Run CICD

Suggested reviewers

  • terrykong
  • yuki-97

Pre-merge checks and finishing touches

✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix: grad norm check for automodel gpt oss nightly' directly and specifically describes the main change—adjusting the grad norm check for the automodel GPT OSS nightly test.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Test Results For Major Changes ✅ Passed PR contains minor test metric threshold adjustments for grad_norm validation, directly supporting prior PR #1693 changes. No major features or breaking changes present.
✨ Finishing touches
  • 📝 Generate docstrings
📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c8d6569 and 7795224.

📒 Files selected for processing (1)
  • tests/test_suites/llm/sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel.sh
🧰 Additional context used
📓 Path-based instructions (4)
**/*.sh

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.sh: Use uv run instead of python to execute scripts
Follow the Google Shell Style Guide for shell scripts

Files:

  • tests/test_suites/llm/sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel.sh
tests/test_suites/**/*.sh

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

tests/test_suites/**/*.sh: When adding support for a new model, create a corresponding driver shell script under tests/test_suites/ in the matching domain
Driver shell scripts should match the YAML base name with .sh extension and invoke training entrypoint with uv run

Files:

  • tests/test_suites/llm/sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel.sh
!(**/tests/**|**/test_*.py|**/test_*.sh)

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Add the NVIDIA copyright header to all Python files and shell scripts (excluding tests). The header should include the current year

Files:

  • tests/test_suites/llm/sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel.sh
**/*.{py,sh}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

The NVIDIA copyright header should appear at the top of all Python files and shell scripts (excluding tests)

Files:

  • tests/test_suites/llm/sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel.sh
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
  • GitHub Check: Lint check
  • GitHub Check: Post submodule check comment / Comment on PR
  • GitHub Check: Post automodel integration comment / Comment on PR
🔇 Additional comments (1)
tests/test_suites/llm/sft-gpt-oss-20b-1n8g-fsdp8ep8-automodel.sh (1)

39-40: Ensure PR description documents empirical justification for gradient norm bounds change.

The gradient norm bounds change (from < 2.5 to [10.0, 17.5]) represents a significant shift in expected training behavior. Per the learnings on convergence-impacting changes, the PR description should include evidence demonstrating that this adjustment is empirically grounded and does not introduce regressions.

Confirm that the PR description includes:

  1. Actual gradient norm values observed during training runs post-PR fix: grad norm calculation for dtensor v2 #1693
  2. Justification for the specific bounds [10.0, 17.5]
  3. Verification that this change does not regress convergence or final metrics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@hemildesai hemildesai added the CI:L0 Run doctests and unit tests label Jan 4, 2026
@terrykong terrykong enabled auto-merge (squash) January 5, 2026 00:00
@yuki-97 yuki-97 added CI:L0 Run doctests and unit tests and removed CI:L0 Run doctests and unit tests labels Jan 5, 2026
@terrykong terrykong merged commit 13c3cd6 into main Jan 5, 2026
69 of 74 checks passed
@terrykong terrykong deleted the hemil/fix-gpt-oss-nightly branch January 5, 2026 06:56
chtruong814 pushed a commit that referenced this pull request Jan 5, 2026
@hemildesai
fix: grad norm check for automodel gpt oss nightly (#1708)
f443763 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
@coderabbitai coderabbitai bot mentioned this pull request Jan 5, 2026
Merged
parthmannan pushed a commit to parthmannan/RL that referenced this pull request Jan 15, 2026
@hemildesai @parthmannan
fix: grad norm check for automodel gpt oss nightly (NVIDIA-NeMo#1708)
71a7fa8 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Parth Mannan <pmannan@nvidia.com>
yuanhangsu1986 pushed a commit to yuanhangsu1986/RL-Nemontron-Edge-Omni that referenced this pull request Feb 12, 2026
@hemildesai @yuanhangsu1986
fix: grad norm check for automodel gpt oss nightly (NVIDIA-NeMo#1708)
590c86e 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: yuanhangs <yuanhangs@nvidia.com>
yuanhangsu1986 pushed a commit to yuanhangsu1986/RL-Nemontron-Edge-Omni that referenced this pull request Feb 21, 2026
@hemildesai @yuanhangsu1986
fix: grad norm check for automodel gpt oss nightly (NVIDIA-NeMo#1708)
ad6bd9e 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: yuanhangs <yuanhangs@nvidia.com>
seonjinn pushed a commit that referenced this pull request Mar 8, 2026
@hemildesai
fix: grad norm check for automodel gpt oss nightly (#1708)
87f0a68 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
seonjinn pushed a commit that referenced this pull request Mar 8, 2026
@hemildesai
fix: grad norm check for automodel gpt oss nightly (#1708)
d1ab7ed 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
seonjinn pushed a commit that referenced this pull request Mar 9, 2026
@hemildesai
fix: grad norm check for automodel gpt oss nightly (#1708)
7d46880 
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@terrykong terrykong terrykong approved these changes

Assignees

No one assigned

Labels

CI:L0 Run doctests and unit tests r0.5.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hemildesai @terrykong @yuki-97